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Perturbations of Jacobi polynomials and 
piece-wise hypergeometric orthogonal systems 

Neretin Yu.A. 1 



We construct a family of noncomplete orthogonal systems of functions on the ray 
[0,oo]; these systems depend on 3 real parameters a, j3, 8. Elements of a system 
are piece- wise hypergeometric functions, having a singularity at x = 1. For 6 = 
these functions vanish on [1, 00) and our system is reduced to the Jacobi polynomials 
P"' 13 on the segment [0, 1]. In a general case, our functions can be considered as an 
interpretation of P"+g- Our functions are solutions of some exotic Sturm-Liouville 
boundary problem for the hypergeometric differential operator. We find the spectral 
measure for this problem. 

1. Formulation of result 

Results of the paper are formulated in Subsections 1.1- 11.51 Next, in 11.61 - 
11.91 we discuss existing and hypotetical relations of our phenomenon with some 
other mathematical topics. 

1.1. Jacobi polynomials. Preliminaries. Recall that the Jacobi 
polynomials P"'' 3 are the polynomials on the segment [—1,1] orthogonal with 
respect to the inner product 



</,<?>= / J(y)g(y)(l-yr(l + yfdy, 



a > —l,/3 > -1 

These polynomials are given by explicit formulae, (see jS], 10.8(16)), 

F 

■n 

F 



P^iv) r(n + Q + 1) 



T(a + l)n\ 
(-l)'T(n + /3 + l) 



n, n + a + (3 + 1 1 — y 
a + 1 ' 2 



(-l)'T(n + /3 + 1) (l-y\- a 



n, n + a + (3 + 1 1 + y 

(3 + 1 ; ^T~ 



F 



r(/3 + l)n! V 2 
Here F = 2-F1 is the Gauss hypergeometric function 

a. b 



n + /3 + 1,—a — n 1 — y 
13 + 1 ; ^T~ 



(1.1) 

(1.2) 
(1.3) 
(1.4) 



F[a,b;c;x] = F 



3=0 



{c)k 



and (a)k := a(a + 1) . . . (a + k — 1) is the Pochhammer symbol. 

The expressions (I1.2JI . (f 1 . T^fl are polynomials since (— n)k = for k > n. The 
last expression (|1.4(l is a series, it can be obtained from (jl.3|l by the transfor- 
mation (see ^(2.1.22-23)), 



F [a, b;c;x]=(l- xf- a - b F[c -a,c-b;c; x] 



(1.5) 



1 Supported by grant NWO-047.017.015 



1 



Norms of the Jacobi polynomials with respect to the inner product 1)1. lfl are 
given by 

pa,fiu2 = , pa.fi po.A = g^T(n + a + 1)I> + g + 1) 

' ™ ' n ' (2n + a + /3+l)n\T(n + a + (3+l) [ ' 



The Jacobi polynomials are the eigen-functions of the differential operator 



(a + P + 2)y)] 



dy 



(1.7) 



Precisely, 



DP%'P = -n(n + a + + 1)P£ ,/J 



1.2. Piece-wise hypergeometric orthogonal systems. Now, fix 9 6 C 
such that 

O^Rc0<l (1.8) 



Also, fix a, /3 € C such that 

-1 < Rea < 1, a ^ 0, Re/3 > -1 



(1.9) 



Consider the space of functions on the half-line x > equipped with the 
bilinear scalar product 



{/,<?}= / /(x)5W(l-x) Q ^dx+ 



sin a + am 



f(x)g(x)(x~l) a x dx (1.10) 



Denote by #(x) the Heaviside function 
#(*) = 



1, x > 0; 
0, x < 



Let p £ C ranges in the set 

p-0GZ, Re(2p + a + /? + 1) > 0, l+p + a^0 (1.11) 
Define the piece-wise hypergeometric functions <&p(x) on the half-line [0, oo) 



by 



r(/? + i) 



r(i + p + a) 

n-p) 



F 



—p,p + a + (3 + 1 
+ 1 

p+a + l,p + a + /3+l_l 
2p + a + f3 + 2 ' x 



H(l-x) + 
x - a -fi- P -i H ^ x _- Vj ( 112 ^ 



Theorem 1 . 27ie functions $ p are orthogonal with respect to the symmetric 
bilinear form 

{$„,$,} = /orp^q (1.13) 
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and 



lit * 1 = T 2 (2p + a + /3 + 2)r(l + p + g)r(p + 1) 

1 p ' pl (2p + a + (3+l)T(p + /3+l)T(p + a + p+l) l ' j 

We can consider also the Hermitian inner product 



l 



(/,<?> = / /(z)ff(z)(l-z) a x p cfe + \ y / /(a:)3(aO(a:-l) t V(fc (1.15) 

sin c/7r 



here we must assume 9, a, [3 € R, and 

< 6> < 1, -Ka<l, /3>-l, 2p + a + /?+l>0 

By our theorem, the functions $ p are orthogonal with respect to the inner 
product l|1.15fl . If the factor sin(a + 0)tt/ sin(oj7r) is positive, then our inner 
product also is positive definite. 

Remark. The system <£> p is not a basis in our Hilbert space. 

1.3. Comparison with the Jacobi polynomials. Let us show, that our 
construction is reduced to the Jacobi polynomials in the case 8 = 0. 

First, assume x = (1 + y)/2 in the formulae (|1.1|) . i|1.3|) . We observe that 
the first summand in (|1.12l) is a Jacobi polynomial. The second summand in 

<|1.12fl is since it contains the factor T(-p)^ 1 . This factor is if p = 0, 1, 2, 

Thus, for an integer p, 

w° rl *w" + V g-w-i 

Hence for 9 = our orthogonality relations are the orthogonality relations 
for the Jacobi polynomials. 

1.4. Singular boundary problem. Consider the differential operator 



d 

' dx 2 v ' ' dx 



D := x(x - 1)— + {ft + 1 - (a + p + 2)x)— (1.16) 



The functions $ p satisfy the equation 

D% = -p{p + a + [3+ 1)% (1.17) 

More precisely, the function <& p is given by different Kummer solutions of the 
hypergeometric equation on the intervals (0, 1), (l,oo), see formulae 4 , 2.9(1), 
(13)). 

Let a ^ 0. 2 Now we define a space £ of functions on [0, oo). Its elements 
are functions f(x) that are smooth outside the singular points x = 0, x = 1, 
x = oo; at the singular points they satisfy the following boundary conditions 

2 Otherwise, below we have logarithmic asymptotics at % = 1. 
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(a strange element of the problem is the condition b); its self-adjointness is not 
obvious). 

a) The condition at 0. A function / is smooth at 0. 

b) The condition at 1. There are functions u(x), v(x) smooth at 1 such that 

!u(x) + v(x)(l - x)~ a . x<l; 
sin for (1.18) 
-— —u(x)+v{x)(x-l) a X>1 
sin(a + 0)7r 

c) The condition at oo. There is a function w(y) smooth at zero, such that 

fix) = w{l/x)x~ a ~ ,3 ~ r ~ 1 for large x 

where r is the minimal possible value of p. 

Theorem 2. a) $ p e £. 
b)Forf,ge£, 

{Df,g} = {f,Dg} 
Obviously, this implies the orthogonality relations for p ^ q. Indeed, 

{D$ p , $ J = {%, = -p{p+a+P+\){%, $ J = -q(q+a+ f 3+l){%, $J 

and hence 3 $ g } = 0. 

Denote by JC the Hilbert space with the inner product (|1.15(l . Obviously, 
£ C DC. 

Theorem 3. The operator D is essentially self-adjoint on £. 

Remark, a) We can replace the boundary condition at oo by the following: 
fix) = for large x. Thus, our complicated formulation is not necessary. 

b) If P ^ 1, then we can replace the condition at by the following: f(x) = 
at some neighborhood of 0. For (3 < 1 the latter simplifying variant gives a 
symmetric, but not-self-adjoint operator 4 . Possible self-adjoint conditions are 
enumerated by points A : fJ, of the real projective line; they can be given in the 
form 

fix) = A(X + nx~P) + x<p(x) + x-P +1 ip(x) 

where ip, tp are functions smooth near 0, A ranges in C. The condition given 
above corresponds to fi — 0. Thus, our requirement of the smoothness is not an 
absence of a condition, it hides a condition for asymptotics. 

1.5. Expansion in eigenfunctions. Our orthogonal system is not com- 
plete; hence our operator has a partially continuous specter. In such a case, 
a usual expansion of a function in a series of the Jacobi polynomials must be 
replaced by the eigenfunction expansion of D in spirit of Weyl and Titshmarsh 
(see El). 

3 Under our conditions for parameters, p(p + a + (3 + 1) = q(q + a + (3 + 1) implies p = q. 
4 for a discussion of difference between symmetry and self-adjointness, see any text-book 
on the functional analysis, for instance, P] 
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For s € R, we define the function ^ s {x) on [0, oo) given by 



9,(x) = F 



2±|±1 + i5) £±|±i _ is 



2r(/3+l 



#(1 - x)- 



•Re< 



r(-2is)cos(^ 



IS 7T 



x F 



Obviously, 



sin(0 + a)7r 
■ i 

1 + 2is 



a+13+l _r_ ■ „ a-/3+l 
2 ' 2 



2 

ZS 1 



r( o±|±l _ j s)r( ^3±i _ 

a .-(o+^+i)/a-fal fl -( a ._ 1 j (i.i9) 



x 



* s (x) = #_ s (:r) 

Remark 1. The both summands of <3/ s are solutions of the equation 

.1 



(1.20) 



Df = -(-(a + [3+l) 2 + s 2 )f 

Indeed, the first summand is the same as above, we substitute p = — 
to (|1.12|) . The hypergeometric function in the second summand is a Kummcr 
solution of the equation (|1.2U|) . Since the coefficients of D are real, the complex 
conjugate function also is a solution of the same equation. 

Remark 2. The functions ^(x) satisfy the boundary condition at x = 1. 

Remark 3. The functions <& p with p — 9 e Z are all the L 2 -eigenfuctions for 
the boundary problem formulated above. Also, the functions ^ s (x) are all the 
remaining generalized eigenfunctions of the same boundary problem. See below 
Section 4. 

This 3 remarks easily imply the explicit Plancherel measure for the operator 

D. 

Consider the Hilbert space V, whose elements are pairs (a(p),F(s)), where 
a(p) is a sequence (p ranges in the same set as above), and F(s) is a function 
on the half- line s > 0; the inner product is given by 



[(a,F);(b,G)] = £ 



i(p)b(p) 



sin 8tt sm(8 + a)iT 

4ttT(P + 1)2 J Q 



r( 



")r(- 



-a+0+1 



is) 



T(2is) cos(^ + 6 - is)ir 



F(s)G(s)ds 
(1.21) 



Wc define the operator Uf i— ► [a p , F(s)) from "K to V by 

a(p) = (f,® P )x, 
F(a) = (f,* a )x 



Theorem 4. The operator U : J£ — > V is a unitary invertible operator. 
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In particular, this theorem implies the inversion formula, 



IJ- 1 = U* (1.22) 

1.6. Discussion: shift of the index n for classical orthogonal bases. 

Thus, for Jacobi polynomials P"'@ there is a perturbated nonpolynomial orthog- 
onal system obtained by a shift of the number n by a real 8. Similar deformations 
are known for several other classical orthogonal systems. In the present time, 
the general picture looks as confusing, and I'll shortly refview some known facts. 

a) Meixner system. Perturbated systems were discovered by Vilenkin and 
Klimyk in see, also, |23, and more details in 

b) Laguerre system, see detailed discussion 7 

c) Meixner-Pollachek system, see. |2(J| . 

d) Continuous dual Hahn polynomials, a (noncomplete) construction was 
obtained in |19) . 

In all the cases enumerated above, the perturbated systems are orthogonal 
bases, indexed by numbers n + 9, where n ranges in Z. 

All the such deformations are obtained in the following way. Allmost all clas- 
sical orthogonal hypergeometric systems 5 arise in a natural way in a detailed 
consideration of highest weight and lowest weight representations of SL/2(K). 
Repeating the same operations with principal and complementary series of rep- 
resentations, we obtain deformed systems 6 . 

All the basic formulae existing for orthogonal polynomials (see lists in [5], [§], 
|21p. "survive" for deformed systems; this confirms the nontriviality of the phe- 
nomen under a disscussion. Moreover, representation-theoretical interpretation 
allows to write such formulae quite easily. 

Note that the systems a)-c) can be partially 7 produced by very simple op- 
erations. 

a) Fix real parameters h, a, t. Consider two orthonormal bases 



e k (z) = z n , f n (z) = ( Zd ?l + Sl *l ) (zsht+chty'^iz^sht+chty 

\ z sh t + ch t J 



l + h+ia 



in the space L 2 on the circle \z\ = 1. Expanding one basis in another one, we 
obtain a matrix, whose rows are orthogonal in the space ^(Z). These rows form 
a Meixner-type system. 

b), c) Consider an orthonormal basis in L 2 on R consistng of functions 



f n (x) = (1 + ix)- n - 1+h+ia {l - ix) n - M 



5 The only possible exception is the Wilson polynomials (the author does not know, are 
they appear in this context). 

6 For the Hahn system, the group SL2(R) does not provide a sufficient collection of param- 
eters, nevertheless this method has an heuristic meaning 

7 Partially, since such bases form "series" imitating "series of representations". Construc- 
tions imitatiting "complementary series", can be hardly observed without representation the- 
ory. 
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n ranges in Z. Applying the Forier transform, we obtain a Laguerre-like piece- 
wise confluent hypergeometric basis. 

Considering the Mellin transform f n (x) i— » (F+ (t) , F~ (£)) , 



F,t(t)= U*V dx, F~(t)= U-*V dx 
Jo Jo 

of the same functions, we obtain a Meixner-Pollachek-like system. It appears 
that this system is an orthonormal basis in a space of C 2 -valued functions on 
M. 

d) Hahn-type system can be obtained from the same functions /„ by applying 
of the bilateral index hypergeometric transform introduced in |19|. 8 

Now, the author does not know nor represetentation-theoretical interpreta- 
tion, nor a way for "simple production" Jacobi-type systems. 9 

1.7. Discussion: multi-contour boundary broblems. Boundary prob- 
lems for systems of contours with cross-glueing of asymptotics are not well 
known. We intend to explain some natural origins of their appearence. 

First, consider a model example, the equation 

d 2 

-f(x,y) = Xf(x,y) 



dxdy" 

where A is a constant. Since, this equation is similar to 
/ d 2 d 2 \ 

{Wx + Wy-) Kx ' v) = mX ' v) 

let us imitate the radial separation of the variables for the Laplace operator. 

Let / be a smooth compactly supported function on the plane. For r > 
assume 

/oo 
/(reSre-V**"^ (1.23) 
-oo 

Respectively, 



1 poo / \ ifJ,/2 

1 / , , , / X N 



^ J-oo \y; 



Our equation now transforms to 



Vg = Xg (1.24) 



where 

4 V dr 2 r dr r 2 

8 Again, there arises a system of C 2 -valued functions. There is a general interesting problem 
on existence of theory of vector-valued and matrix-valued special functions. Two examples 
are given by us just now. 

9 The specter of the operator D is the same as of a tensor of a pair of representations of 
SL2(R) that are contained in principal and complementary series. Possible, this coincidence 
is not a chance. 
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Figure 1: 







a) r 'I b) 

a) Orbits of the group of hyperbolic rotations 

b) The space of orbits in general position 



Dr^ - 






I_ Br -in 


Dr ifi - 









c) 

c) Asymptotics of 4 functions. 

d) Four contours for the Bessel operator D. 



oo . 



d) 



DO 



For a fixed //, we obtain a Sturm-Liouville problem for the operator D. 
Next, for a fixed // ^ the function ^(r, /z) has the following asymptotics 

g(r,n) ~ Ar^ + Br~ ifl , r -> 0, 

where 

/-oo 

A=-rH/i/2)/(0,0) + ^ (/(x,0)-/(0,0)e- a;2 )a ; -^- 1 d a ; 

and £? is given with a similar expression. 

In the integral transform (|1.23|l only the values of / in the quadrant x ^ 0, 
y ^ take part. Hence, consider functions f(—x,y), f(—x,—y), f(x,—y) and 
construct 3 more functions (|1.23l) . As a result, we obtain 4 functions, whose 
asymptotics are shown on Fig.lc. But it two cases we obtain the equation 
11.24fl transformed by A i— > —A. Ncthertheless, the same result can be obtained 
by the substitution r i— > ir in the Bessel operator D. Now, in all the 4 cases 
the operator D became being the same, but the argument r now is contained in 
different contours. 

As a result, we obtain a boundary problem of the following type (see Fig. 
l.d): we have the Bessel operator D defined on quadruples of functions, each 
function is defined on its own contour, and the asymptotics at zero satisfy certain 
condition of cross-gluing. 

Emphasis, that our actions were completely standard. Namely, our equation 
is invariant with respect to action of the group T of hyperbolic rotations (x, y) i— > 
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(xe t ,ye^ t ) of the plane, see Fig. la). We simply use this invariance for the 
separation of the variables. But the set of orbits of T in a general position on 
R 2 is disconnected and consists of 4 components (see Fig. lb). This produces 
4-contour problem. 

More interesting example is the problem on a spectral decomposition of a 
SL 2 (K)-invariant Laplace operator on the torus \z\ = 1, |u| = 1, 

d 2 1 ~ d 1 d 

A = —{z - u) 2 —— + -{Ou + tz)(z - u)— + -{Oz + tu){u - z)—+ 
azou u oz z ou 

H (0U + tz){6z + tu) 

zu 

The operator is self-adjoint in L 2 on the torus 10 in the case 

Re(6 + r) = 1, Im 6 = Im r, Re(9 + t) = 1, Im 6 = Im r 

This operator is an interesting, complicated and not well-undertand object. 
It was concidered in several works , [HI, [0]], |7j). We intend to discuss 
some details that are absent in these works. 

The group SL2(M) acts on the circle by Mobious transformations, and hence 
it acts on the torus since it is a product of circles. Consider an one-parametric 
group r C SL 2 (IR) and separate variables using T as in the previous example. 

There are 3 possibility 

a) r = K is a subgroup of rotations of the circle. 

b) r = P is parabolic, i.e. it is an one-parametric subgroup having one fixed 
point on the circle. 

c) r = H is hyperbolic, i.e., T is an one-parametric subgroup having two 
fixed points on the circle. 

Separations of variables corresponding to these subgroups is reduced respec- 
tively to Forier expansion, Fourier transform, and Mellin transform (a circle is 
a real projective line, and then we can Fourier and Mellin transform). 

As a result, we obtain 3 variants shown on Fig. 2 (we represent the torus as 
a square). 

Numerous nonstandard boundary problems of this kind (in particular, multi- 
dimensional) arise in a natural way 11 , in the non-commutative harmonic anal- 
ysis (the topic apprently arises to [2"5]V 

Actually, they are not well-studied. Several one-dimensional problems for 
the Legendre equation were examined by Molchanov (2], JH1> J^j and 

Faraut |>I for obtaining of the Plancherel formula on rank 1 pseudo-Riemannian 
symmetric spaces (in particular, this class of spaces includes multi-dimensional 
hyperboloids). 

10 There are several more cases of self-adjointness in other functional spaces 

11 Miller's treatise 1121 contains lists of various separations of variables for several classical 

partial differential equations. Some of such ways produce multi-contour problems, in the book 

this is not mentioned. 
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Figure 2: 

oo 




a) The diagonal of the torus is the singular line of the operator A. Orbits of K 
are circles parallel to the diagonal. The space of orbits is a circle, but one of 
point of this circle is distiguished. 

b) After a separation of variables, we obtain a hypergeometric operator on the 
contour Imx = 1/2 (it can be replaced by \x\ = 1). 








c) The group P has one fixed point on the torus (it is the corner point on our 
figure). The space of orbits in a general position consists of two components. 

d) A separation of variables produces a confluent hypergeometric differential 
operator(in some cases, a bessel operator) on the pair of contours oo. The 
diagonal of the torus corresponds to the point 0, and the boundary of the square 
to the point oo. 

oo 




oo 



e) The group H has 4 fixed pointson the torus. The set of orbits in a general 
position consits of 6 components. 

f) A separation of variables produces a hypergeometric differential operator on 
6 contours. Its singular points correspond to the diagonal of the torus and the 
pieces H, X of the Fig. e). 

It os more reasonable to think, that we have 3 contours (oo,0), (0,1), (l,oo) 
and C 2 -valued functions on each contour. 
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Remark. Examples enumerated above allows to think that reasonable 
multi-contour problems has approximately the following form. We consider 
the space of all the functions as a module other the space of smooth functions. 
Domain of definiteness of a PDE system consists of functions, that are contained 
in a fixed (explicetly defined) submodule of our module. A possibility of explicit 
solution of such problems looks as questionable; but also there are no reasons 
to think that roundabot ways are better. 

1.8. Discussion. Degree of rigidity of the problem. There are 
two classical variants of expansion of the hypergeometric differential operator 
in eigenfunctions. One case gives the expansion in the Jacobi polynomials. 
Another one gives the Weyl-Olevsky index hypergeometric transform 



9(s) = 

1 



o 



/(*) 2 F l \ h + is > b - 13 ; - x ] x b+c-i (i + x f+c-i dx 
V b + c J 



L 



(^■^-('^)^)-^(^. r (t+ s +i8) r) 



(1.25) 

Remind (see, for instance, jSJ, Chapter XIII), that this transformation is an 
unitary operator 

r(b + is)T(c- 
T(2is) 

Unitarity condition implies the inversion formula ljl.22jl . This transformation 
is an object interesting by itself having numerious applications in harmonic 
analysis and theory of special functions (see |27| . |17|. |18j). 

There is the third variant of this transformation recentely obtained in |19|. 
it corresponds to the contour shown on Fig. 2b). 

Our Theorem 4 can be interpreted as one more (but strange) analog of 
expansion in Jacobi polynomials. Evidentely (see, for instance our subsection 
ll.7|l . there are other analogs having natural origins. 

On another hand, we can assign arbitrary multiplicities to the contours (0, 1), 
(1, oo), (oo, 0), on C; after this there arise a wide (and even too wide) freedom 
to invent boundaru conditions as in l|1.18(l . 

As a model example, consider the same operator D defined in L 2 (0, 1) with 
respect to the same weight ar (1 — x) a . If a > 1, (3 > 1, then the space 2)(0, 1) of 
compactly supported smooth functions on (0, 1) is a domain of self-adjointncss. 
If 

-1 < (3 < 1, -1 < a < 1 

then the deficiency indices of D on 2)(0, 1) are (2, 2). In fact, the both solutions 
of the equation Df = Xf are in L 2 for all A. 

Fix n, v £ K. Let us write the boundary conditions 

f(x) =A[1- ^y^x~P] + Xip Q {x) + x-P +1 Mx) near x = 

f(x) = B\l + ^^7^t(1 - x)- a ] + xipxix) + (1 - x)~ a+1 Mx) near x = 1 
1 (—a) 
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where ipo, ipo are smooth near x = and tp±, ip\ are smooth near x = 1. 

Then the specter is discrete and A = — p(a + j3 + 1 + p) is a point of the 
specter iff 



1 



A 



T(f3 + p+l)T(-p-a) T(p+ l)T(-u-(3-p) 

A 1 



r(-p)r(p + a + (3 + 1) T(-P-p)T(a+p+l) 

The equation seems nice, but apparently it is nonsolvable. 

In any case, not all the boundary conditions have the equal rights. 
1.9. Discussion. An attempt of an application. Denote 12 



= (1 - xYH{\ - x), 



Equating 



we obtain the following identity 



7r sin07rsin(0 + o)ttx 



r( 



a+/3+l 



w)r(- 



»«)r(2i|+i + e + is)r( - a -f +1 



a+/3+3 



is)T(y 



o-M+3 



+ is) 



sin(0 - /i)7r sin(0 - v)w J^(2p + a + /3 + 1) 



„ p + a + f3 + l,p + (3 + 1, — v-\-p, — /i + p 

X [p + a + l3 + n + 2,p+a + /3 + v + 2,p+a+l,p+l 

T(f3 + l)r(a + fx + i/ + 1) 

r(a + /3 + n + v + 2)r(/i + l)T(v + l)r(a + /i + l)r(a + i/ + 1) 



(1.26) 



This identity is a kind of a beta-integral, continuous and discrete beta-integrals 
are well-known, see Our integral has a mixed continuous-discrete form, here 
an integral and a countable 6-F5-SUH1 are present 13 . 

Under the substitution fj, = — 1 to i|1.26|) . the g-Ps-sum vanishes, and we 
obtain the following beta-integral obtained in |18|. 



1 

2^ 



T(2is)T(b + is) 



a 1 -a 2 - a s ) T\i<k<i<3 r K + a 



nti^-a,) 



12 We imitate a simple way to derive De Branges-Wilson integral, proposed by Koornwinder, 
see, for instance, 1181 . 

13 Analytic continuation of beta-integrals with respect to parameters can produce a finite 
collection of additional summands at the left-hand side (due residues). Our integral has 
another type 
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The substitution 6 = 0, kills the integral term and we obtain a known 
summation formula of s-FVtype. In fact, this is the famous Dougall 5_ff5-formula 
(see, for instance, [I].) 



E 



a + n sin 2ira T(a± + a-2 + 03 + a± — 3) 



IlLi r ( a j + a + n ) r i a j -OL-n) 2tt rii<i<fe<4 r 0j + a k - 1) 



where one parameter is killed by the substitution a± = a. 

It is interesting that our integral does not mojorize the Dougall formula. Ap- 
parepntly, this means that our construction must contain additional parameter 
or parameters. 

1.10. Structure of the paper. In Sections 2 and 3, we give two proofs of 
the orthogonality relations. In Section 3 we also discuss our boundary problem. 
In Section 4 we obtain the spectral decomposition of D. 

Acknowledgments. I am grateful to V.F.Molchanov for a discussion of 
this subject. 

2. Calculation 

We use the notation 

!>!)... I> fe ) 



r 



Oi, . . . , fflfc 

bi, ...bi 



r(&!)...r(6,) 



2.1. The Mellin transform. For a function / defined on the semi-line 
x > 0, its Mellin transform is defined by the formula 

/>oo 

= / f(x)x s dx/x (2.1) 

In the cases that are considered below, this integral converges in some strip 
a < Re s < t. The inversion formula is 

7+ioc 



1 i-j-t-ioo 

f{x) = — / Tlf(s) 



7 — 200 

there the integration is given an over arbitrary contour lying in the strip a < 
Re s < t. 

The multiplicative convolution / * g is defined by 

/>oo 

f*g(x)= / f(x)g(y/x)dx/x (2.2) 
Jo 

The Mellin transform maps the convolution to the product of functions, 

<m[f*g](s) = m(s)-Wlg(s) (2.3) 
(if Wtf(s), Wlg(s) are defined in the common strip). 
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2.2. A way of proof of orthogonality. We write two explicit functions 
%i(s), %2(s) and evaluate their inverse Mcllin transforms Ki, Next, we 
write the identity 

K!*K 2 {1) =Wl- 1 [X 1 X 2 ](l) 

and observe that it coincides with the orthogonality relations for $ p and Q q . 

Below, in 2.7 we explain an origin of the functions K\, K 2 . The calculation 
on formal level (without following of convergences, conditions for convolutions 
theorem etc.) is performed in the next Subsection 2.3. In Subsections 2.4-2.6, 
we follow omited details. 

2.3. Evaluation of the convolution. We use the following Barnes-type 
integral E2, 8.4.49.1) 



2m 



c, 1-6 
a 



-Hoc 



s, a — s 
s + 1 — 6, c — s 



= F 



a, b 



;x 



c,l-b 
c — a, 1 + a 



x s ds — 
H(l-x) + 
F 



a, 1 + a 
l + o- 



-c 1 
b 5 x 



H(x-1) (2.4) 



where x > 0. The integrand has two series of poles 

s = 0, — 1, — 2, . . . , s = a,a+l,... 

The integration is given over an arbitrary contour lying in the strip < Re s < 
Rett (such contour separates two series of poles). The condition of the conver- 
gence is Re(c — a — b) > —1. 

Remark. A reference to the tables of integrals is not necessary, since the 
identity can be easily proved using the Barnes residue method, see, for instance, 

(El, ED). 

We consider two functions 3Ci(s), %2(s) given by 



DCi(s) = r 
3C 2 (s) :=T 



r 



p+l,p + a+l 
(3 + P+l 

2q + a + 13 + 2, -a - q 
q + a + (3 + 1 



s,(3 + p+ 1 - s 
s + p + a+l,/3 + l-s 

s + a + q,/3 + l — s 
s, q + (3 + 2 - s 



■ r 



we assume that p — 6, q — 9 e Z. Using formula J22J, we evaluate their inverse 
Mellin transforms, 



K x {x) :=F 



P + /3 + 1,— p — a 



(3+1 



;x 



(3+ l,p + a + 1 
-p,2p + a + (3 + 2 



H(l-x) 
■ F 



(3+p+ l,p+ 1 1 
2p + a + (3 + 2 ' x 



H(x-1) (2.5) 
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K 2 (x) =x q+a F 



q + a + (3 + l,q + a + l_ 
2q + a + f3 + 2 ' 

2q + a + (3 + 2, -a - q 
q + 1,0 + 1 



F 



H(l-x)+ 

q + a + (3 + 1,-q 1 
0+1 ' x 



F(ai - 1) 



Now we write the identity 

K x *K 2 (l) =9Jl- 1 [0C 1 3C 2 ](l) 
and multiply its both sides by 



r 



2p + a + (3 + 2, q + 1 
/3+1,-a-g 



We obtain the following identity 



2p + a + (3 + 2, q + l 
/3 + 1, 



-a — q 



2q + a + f3 + 2, 



-a — q 



q+l ,13+1 



x / F 
'o 



p + (3 + 1, —p — a 
(3 + 1 



X P +1 F 



q + a + (3+ 1,-q 
P + l 



(2.6) 



dx/x+ (2.7) 



2p + a + (3 + 2 




f3 + l 


,-a- 



(3 + 1 ,p + a + l 



-P, 



2p + a + (3 + 2 



x / x 



-P-p-Xp 



(3 + p + l,p + l 1 
2p + a + (3 + 2 1 x 



X - q - a F 



q + a + fi + l,q + a + l 1 
2q + a + (3 + 2 ' x 



dx 
x 



(2.8) 



2m 



2/ 


+ a + 


P + 2,q + l 


r 


(3+1 


,p + a + 1 


r 


2q + a + (3 + 2, 


—a — q 




(3+1 




—a — q 


/3H 


-p+1 




q + a + (3 


+ 1 



n / r 

-zoo 



H, /3 + P + 1 



s +p + a + 1, + 1 -s 



s + a + q, 



[3+1-s 



0, q + (3 + 2 - s 



(2.9) 



Remark. Sixteen boxed T- factors are intensionally are not canceled. One of 
decisive element of the calculation is cancelations in the row (|2.9|l . But trick 
with changing the integer parameters to, n by the shifted real parameters m + 9, 
n + 6 from 2.7 guarantee this cancelation. □ 
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We must identify this identity with the following orthogonality identity for 



2p + a + (3 + 2 
(3 + 1 
fi 



2q + a + (3 + 2 
(3 + 1 



F 



p + a + (3 + 1, — p 
(3 + 1 ' 



F 



q + a + (3 + 1, — q 
(3 + 1 ' 



x> 3 (l-x) a dx+ 

(2.10) 



sin(a + 6)n 



r 



sm t/n 

oo 



1 + p + a, 1 + q + a 

-p, -q 



x / F 



p + a + j3 + l,p + a + ll 
2p + a + (3 + 2 ' x 



q + a + (3 + l,q + a + ll 
2q + a + (3 + 2 ' x 



x x- 2a -P- p - q - 2 (x - 1)" dx = (2.11) 



~ 2p + a + (3+l 

where <5 P - ? ,o is the Kronecker symbol. 

We identify $Fft-$TQ with l|£Tn jl -l|2~T3 J) line-by-line 

1. The summand W7w equals the summand I2.1fl\) . We transform the first 
F-factor of the integrand by (|1 . B|> . 

2. The summand \2.tfy equals the summand \2.11\l . First, we transform the 
first F-factor of the integrand by i|1.5fl . Secondly, we apply the reflection formula 
r(z)r(l — z) = nj sin(7rz) to the gamma-product in Q2.8|) . 

r(l + a + g)r(l + a+p) sin(l + a + q)n 
r(- J3 )r(-q)sin(l + g ) 7 r 

Next, we use n := q — 9 £ Z, 

sin(l + a + q)n _ sin(l + a + n + 8)n _ (-l) n+1 sin(a + &)n 
sin(l + <?)7r ~~ sin(l + n + 6)n ~ (-l) n+1 sin(6»)7r 

Remark. After this T-factors in l|2.7|l . (|2.8|l transforms to the form 

const ■u(p)u{q), const -v(p)v(q) 

with constants that do not depend on p, q. This is necessary for a possibility 
to interprete the identity (|2.7I) (12.91) as an orthogonality relation of single type 
functions $ p , $ g . Certainly, this was achieved due a multiplication of our iden- 
tity by a T- factor (|2.6I) . But, a priory, a possibility of such multiplications is 
not obvious; as far as understand this is not predictible beforehand. □ 



2p + a + (3 + 2,2p + a + (3 + 2,1 + p + a,p + 1 
p + (3+l,p + a + (3+1 



(2.12) 



-l,P + a + l 
-a-q,-p, 
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3. Right-hand sides, i.e., $2.y\) and \2.12}) . We apply the Barnes-type inte- 



gral 



1 

2ni 



a + s,b — s 
c + s.d — s 



ds = T 



a + b,c + d — a — b — 1 
c + d — 1, c — a, d — b 



see E2, 2.2.1.3 14 and obtain 



1 

2tH 



0+1+p— s, a + q + s 
l+p + a + s,q + f3 + 2-s 



ds = 



r 



a + P+p + q + l, 1 
a + P+p + q + 2, q— p + l,p — q + 1 
1 

{a + f3 + p + q + l)T(q -p+ l)T(p -q + 1) ~ 

1 sin(g — p)ir 



(2.13) 



{a + f3+p + q+ 1) n(q-p) 

Since q — p 6 Z, the latter expression is zero if p ^ q. 

Remark. This place, finishing the calculation, can look like misterious. But 
this almost predictible under the point of view proposed in 2.7. Otherwise, how 
can the Jacobi polynomials find a possibility to be orthogonal? 

2.4. Convergence of the integrals. 

Lemma. Under our conditions \l.yp . the integral 



□ 



$p(x)V(l- x) a dx + 



is absolutely convergent. 
Since 



sin(a + 9)ir 
sin ( 07r) 



%{xfx p {x-\) a dx (2.14) 



1 



\%{x)$> q {x)\^^{\%(x)\* + \$ q (x)\*) 
this lemma implies also the absolute convergence of 

1 oo 



1>Jx)^Jx)x^(l - x) a dx + sm ( a + e Jl f §J x )§Jx)x {x-l) a dx (2.15) 

sin(07r) J 



Proof. To follow the asymptotics, we use one of the Kummer relations (see 
01,2.10(1)), 



F 



a. b 



F 



T(c)T(c-a-b) 
T(c-a)T(c-b) 
T(c)T(a + b-c) 

+ r(#j 



a, b 
a + b - c+ 1 



1-z 



il-z) 



c—a — b 



F 



c — a,c — b 
c — a — b + 1 



1-z 



(2.16) 



14 This identity is a partial case of 12.41 . We substitute x = 1 to 12.41 and apply the Gauss 
summation formula for F[a, b; c; 1]. 
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The function <£> p is continuous at x = 0, and hence the condition of the 
convergence of the integral is Re (3 > — 1 . 

The formula l|2.16[> gives the following asymptotics of <J> P as x — > 1 — 

d +C 2 {l-x)- a (2.17) 

For Rea > we have $ p ~ (1 — a;) _Q , and the condition of convergence of 
<|2.14[1 is Rea < 1. For Rea < 0, the function $ p has a finite limit at 1 + 0, and 
the condition of convergence (|2.14|) is Rea > —1. 

Considering the right limit at 1, we obtain the same restrictions for a. 

Obviously, 

<Z> p (x) ~ x-a-P-P-i, x^oo 

Thus the condition of the convergence is Re(2p + a + (3 + 1) > 0. 

We also must avoid a pole in Q1.12p . and this gives a+p+1^0. □ 
Denote m = p — 9, n = q — 9. 

Lemma. For fixed m, n, the integral }2.15\) depends holomorphicaly on a, 
(3, 9 in the allowed domain of parameters. 

Proof. For each given point (cto, 0o,Oo), the convergence of our integral 
is uniform in a small neighborhood of (ao,/3o,#o) (since our asymptotics are 
uniform). It remains to refer to the Morera Theorem (if each integral over 
closed contour is 0, then the function is holomorphic). □ 

2.5. Restrictions necessary for our calculation. First, we used the 
Mellin transform, and hence our functions K\, K 2 must be locally integrable. 
The unique point of discontinuity is x = 1 We have 

K^x) - A l + Af(l - x)~ a , x^l±0; 
K 2 (x) - Bi +Bf(x- 1) Q , x->l±0; 

This implies | Rea| < 1. 

Second, we use the convolution theorem for the Mellin transform. 
The Mellin transform 1)2.1(1 of K\ absolutely converges in the strip 

0<Rcs<,3 + p+ l 

The Mellin transform of K 2 absolutely converges in the strip 

-a-q<Re<(3+l 

We can apply the convolution theorem l|2.3|) if the following conditions are sat- 
isfied 

< f3+p+ 1 
< (3 + a + q + 1 

< (3 + 1 1 — nonemptyness of intersection 

<p + q + a + (3 + 1) of strips 



nonemptyness of strips 
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This domain is nonempty, but it is smaller than the domain of convergence of 
H2.14f> . But the orthogonality identities l|1.13fl . i|l-14[) have holomorphic left- 
hand sides and right-hand sides. Hence they are valid in the whole domain of 
the convergence. 

2.6. Restrictions for 8. These restrictions <|1.8fl were not used in proof. 
In fact, 8 is defined up to a shift 8 t—> 8 + 1. This shift preserves the orthogonal 
system $ p but changes enumeration of the basic elements. 

By this reason I'll explain how the functions %\, %2 were written. 

2.7. Comments. The origin of the calculations. Now, we intend to 
explain the origin of K\ , K2 ■ 

The orthogonality relations for the Jacobi polynomials P"'^ are well known 
but not self-evident. Let us trying to prove them using the technique of Barnes 
integrals, see [TTj . 

Our problem is an evaluation of the integral 



2-^1 



-n, n - 

a- 



a 
■1 



■0. 



2F1 



-m, m + a + (3 
(3+1 ' 



x {l-x) a dx (2.18) 



Denote 



Lx{x) = {l-x) a 2^1 
L 2 (x) = x- - 1 2 F X 



-m, m + a + (3 



H{l-x) 



n,n + a + j3 1 
13+1 'x 



(3 + 1 

H(x-l)+r(x)H(x-l) (2.19) 



where r(x) is an arbitrary function, but we will choose it later. 

Our integral l|2.18|l is the convolution L\ * L2(x) at the point x = 1. We 
intend to evaluate it using the Mellin transform. 

The Mellin transform of L x is (see [22;, 8.4.49.1) 



9RLi(s) = r 



(3 + l,a + m + l 
(3 + m+l 



s, (3 + m + 1 — s 
- m + 1 + s,f3 + 1 - s 



Then we find a function of the form l|2.19|l in the table of inverse Mellin trans- 
forms (see E2J, 8.4.49.1). 15 
We can assume 



ML 2 (s) = r 



2n + a + P + 2, -a - n 
n + a + (3 + 1 



a + n + s, [3 + 1 — s 
s,n + (3 + 2- s 



and after this the desired calculation can be performed. 
After this we change m — > m + 8, n — ► n + 8. 

2.8. Comments. Evaluation of summands in (12.101) 



!7lT|) . Our 

orthogonality relations contain a sum of two integral over different intervals. It 
is ineresting to evaluate each summand separately. 

15 In fact, tables of integrals are not necessary here, since we must write a Barnes integral 
defining a given hypergeometric function on [0, 1], and it is more-or-less clear how to do this. 
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Let p, ijeC. Let us evaluate 

X:=f $ p (o;)$ 9 (x)a^(l-x) a da;, 
■/ o 

Denote 



Y := 



<f> p (x)<f> q (x)x (x-l) a dx, 



a(p, q) 



b(p,q) :=T 



1 



p + q + a + (3 + 

q + l,p + a + 1 
p + /3 + l,g + a + /3 + l 



-r[2p + a + (3 + 2, 2q + a + (3 + 2] 



We write the equation (|2.7(l - (|2.9|l and the same equation with transposed p, q 

sin(a + g)7T , sin(g — pW 

X + ^ ^-Y = a(j>,q)b(p,q)- 



X 



sin g7r 
sin(a + p)tt 



Y = a(p, q)b(q,p) 



(q-p)ir 
sin(q — p)n 



smp7r " ' (q — p)ir 

It is a linear system of equations for X and Y. Its determinant is 
sin(a+p)7r sin(a + q)ir sin air sin(q — p)it 



Hence 



Y = a(p, q) 
X = a(p, q) 



smp7r 



smp7r sin qir 
n(q — p) sin cot 

sinp7r sin qir 
ir(q — p) sina7r 

Tva(p, q) 



sin qir 

b(q,p) - b(p, q) 
sin(a + p)ir 



sin pir sin qir 



(2.20) 



smp7r 



b(p, q) -^±^b(q,p) 



sin qn 



1 



(q - p) sin cot T [p + (3 + 1 , q + a + (3 + 1 , -q, -p - a] 

1 

r[q + /3 + l,p+a + P + l,-p,-q-a] 
3. The boundary problem 



(2.21) 



In this section a^O. 

3.1. Symmetry of the boundary problem. We consider the hypergeo- 
metric differential operator D given by l|1.16[) and the boundary problem for D 
defined in Subsection II. 41 We intend to prove the identity 



{Df,g} = {f,Dg}, f,g€E 



(3.1) 



Let 



H = a(x)^ + b(x)j- 
ax ax 
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be a differential operator on [a, b] formally symmetric with respect to a weight 
i.e., for smooth /, g that vanish near the ends of the interval, 

b rb 

Hf(x) ■ g(x) dx = f(x) ■ Hg(x) dx 



Equivalently, (a//)' = bfi. Then for general /, g, we have 



b rb 

Hf(x)-g(x)dx- f(x)-Hg(x)dx 



{f'(x)g(x)-g'(x)f(x)}a(x)»(x) 



(3.2) 



We apply this identity to the operator D and to the segment [a, b] = [0, 1 — e]. 
Let on some segment [1 — h, 1] we have 

f(x) = u{x) + (1 - x)~ a v{x), g{x) = u(x) + (1 - x)- a v{x) 

with smooth u(x), v(x). Then the correcting term l|3.2f> is 



det 



u'(x) + (1 - x)- a v'(x) - a(l - x)- a - 1 v{x) u(x) + (1 - xY a v{x) 
u'\x) + (1 - x)- a v'\x) - a(l - xy^vlx) u\x) + (1 - x)~ a v(x) 



x x 0+1 {l-x) 



Q + l 



(3.3) 



X — 1 — 6 



The last factor gives the power e a+1 ; recall that — 1 < a < 1. The summands 
of the determinant have powers 

1, E~ a , e- 2a , e-*- 1 ,e- 2a - 1 

But the term with e" 2Q_1 in the determinant is 

det (~ av S x ) < x )\ = o 
\—av(x) v(x) I 

Hence the leading term of the determinant has the order e~~ a ~ x and only 
this term gives a contribution to the limit as e — * +0. Finally, 



lim 



(Df(x)g(x)- f(x)Dg(x)) x (x-l) a dx = «(!)«(!) -«(l)u(l) (3.4) 



For x > 1, we have 

m = -^^-u(x) + (l-x)- a v(x), g(x) 
sm(a + u)ir 



sin 07r 



sin(a + 8)tt 



u(x)+(l-xy a v(x) 
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In a similar way, we obtain 
sin(a + 6)tt 



lim . 

e^+o sm On 



1+e 



(Df(x)g(x)-f(x)Dg(x)) x p {x-l) a dx = -u(l)v(l)+v(l)u(l) 



This finishes the proof of the identity ((3.1(1 

3.2. Verification of the boundary conditions for <E> p . Let us show 
that & P (x) satisfy the boundary conditions at x = 1. It is given by a direct 
calculation, below we present its details. 

We need in expressions for $ p having the form 



Ax{p;x) + B 1 {p;x){l-x) a , x<l 
A 2 {p,x) + B 2 {p,x){x - l)~ a , x>l 



(3.5) 



We intend to expand $ p in power series at x = 1, on the semi-segments 
(1-e.l], [1,1 + e). 

We use the formula 1(2.16(1 for the left semi-segment and obtain 



$ P (x) = r 



2p + a + (3 + 2 
13+1 

+ l,a 
-p, p + a + (3 + 1 



for x < 1 

Next we use the identity 



(3+1, -a 
p + (3 + 1,—p — a 



F 



F 



p + (3 + 1, — p — a 
1 - a 



; 1 — x 



-p,p + a + (3 + 1 
a+1 ' 

(l-x)- a ) (3.6) 



1 - x 



a, b 1 

c ] x 



= r 



c, c — a — b 
c — a.c — b 



F 



a, a + 1 — c 
a + b + 1 - c' 



c, a + b — c 




a, 6 


F 





c - b, 1 - b 
c + 1 — a — 6 



1 - x 
1 - x 



x 



x a (x- 1) 



c— a— 6 



(3.7) 



(this formula is a modified variant of 0], 1.10(4). We obtain 
$ p (x) = T [p + a + 1, -p] x 



x 



2p + a + /3 + 2,-a 
p + l,p + (3 + 1 



P + a + (3 + 1, —p 



2p + a + (3 + 2, a 
p + a + !3 + l,p+a + l 



a + 1 

P + /3 + 1,— p— a 
1 - a : 



1-x 



1-x 



(x-l)- a \ (3.8) 



for x > 1. 

The expressions (|3.6ll . 1)3.8(1 are the desired expansions 1)3.5(1 . We observe, 
that 

sin(p + a)n 



Bi(p,x) = B 2 (p,x); Ai(p,x)/A 2 (p,x) = 



sm p7r 
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We have 

sin(p + a)7r sin(0 + a)ir 

: = : 7, 

8in pn sinew 

and this implies our boundary condition. 

Remark (it will be important below in Subsection l4.2fl . The property (|3.9|l 
is valid iff p — 6 6 Z. Indeed, the difference between the left-hand side and the 
right-hand side is 

sin air sin(# — p)ir 
sinpir sin 9tt 

3.3. Another proof of the orthogonality relations. For p ^ q, the 

orthogonality follows from the symmetry condition l|3.1|) . 
Let us evaluate 

X:=f $ p (x)<f>q(x)xP(l - x) a dx 
Jo 

We preserve the notation l|3.5Jl . By formula QH.4JI . 

lim |^ D%{x)^ q {x)x^{l-x) a dx- ^ P (x)-D^ q {x)x> 3 {l~x) a dx^ = 

= A(p, l)B(q, 1) - A(q,l)B(p,l) 

The constants A(p,l) etc. are the r-coefficients in (13.611 and (|3.8f) : thus the 
right-hand side is known. 

Since <& p are the eigcnfunctions (see l|1.17|) h the left hand-side is 

[~p(p + a + /3 + l)+q(q + a + (3+1)] -X = (q - p)(q + p + a + f3 + l)X 

After simple cancellations we obtain the expression l|2.21|l . 
In the same way, we obtain the expresion f2~2~nf for f™. 

Now we verify our orthogonality relations via a direct calculation. But this 
again is long. 

4. The spectral measure 

Now we intend to evaluate the spectral measure for the operator D in the 
Hilbert space "K using Weyl-Titchmarsh machinery, see 0- 

To avoid logarithmic asymptotics, we assume a ^ 0, =/= 0. 

4.1. Eigenfunctions of the adjoint operator. Now we intend to discuss 
the adjoint operator D* for D. 

Denote by Dom(A) the domain of definiteness of a linear operator A. Recall 
that / € IK is contained in Dom(Z?*) if there exists a function h G J£ such that 
for each g € Dom(D) we have 

(f,Dg) = (h,g) 

In this case, we claim h = D* f. 
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Since D is symmetric, we have 

Dom(ir) D Dom(D) = £ 

Description of Dom(Z?*) is not an important question, really it is necessary 
only description of cigcnfunctions of I?*. 

Lemma. Let S be an eigenfunction of D* . Then 2 satisfies to the boundary 
conditions a), b) at x = and x = 1 from \T~l\ 



Proof. The condition at 0. Let D*E = AS, represent A as 
A = -p(p + a + (3+ 1) 



(4.1) 



There are two solutions of the hypergeometric equation Df = A/ near x = 0; if 
f3 is not a non-negative integer, then they are given by 



Si = *„(a:) = F 



— p,p + a + (3 + 1 



S 2 = x~ p F 



0+1 

—[3 — p, a + p + 1 
l~(3 ' 



(4.2) 
(4.3) 



If j3 5* 1, the second solution is not in IK, and the statement is obvious. 16 
Let — 1 < p < 1. Let / S £, i.e. / is smooth near 0. Expand our eigenfunc- 
tion S as 

5 = u(x) + x~ p v(x), u(x), v(x) £ C°° 

(in fact, u{x) and v(x) are the hypergeometric functions defined from 14.2|l . 
H4.3fl up to scalar factors. If S is in Dom(D*), then 

(Df,E)-{f,D*E) = (4.4) 

Repeating the calculation of Subsection l3.ll we obtain that this difference is 

f(0)v(0) 

Since / is arbitrary, then v(0) = 0. But v(x) = const -F[— (3— p, a+p+l; l—f3; x], 
we have const = 0. 



The condition at x = 1. A proof is similar. A priory, we know that 
E(x) = 



u_(x) + v-(x)(l -x) a , 
u + (x) + v + (x)(x - iy a , 



X < 1 

x < 1 



In fact u± and v± are the hypergeometric functions in the right-hand sides of 
l|3.6fl . (|3.8|) up to constant factors. 

ie For integer (3 > 0, this also is valid. 
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a(x) + b(x)(l-x)- a , x<l 
sin Btx , , , _ , 

: o(x) +b(x)(x- 1) Q , 35>1 



sin(a + 8)ir 

here a(a;), are smooth near x = 1, 

If H S Dom(Z?*), then the condition l|4.4ll is satished. Repeating the consid- 
erations of Subsection 13. II we obtain that (|4.4|l is equal to 

Wl)M1) _ _ 5^ ( _f^i_ o(1K(1) _ i(1)tl+(1)) 

It is zero for all a(l), 6(1) and hence 

sin07r , . 

«_(1)=«+(1, u+1 = — - — — u_(l 

sm(a + yjTT 

But a priory we know v± and u± up to constant factors, and hence, and this 
implies our statement. 

4.2. L 2 -eigenfunctions of D* . 

Lemma. J/He JC is on eigenfunction of D* , then 3 = $ g wii/i q S + Z. 

Proof. Let A be an eigenvalue, let p is given by 1)4. 1|) with Rep > — (a + 
P+l)/2. 

Due the boundary condition at 0, we have 

Z = const F{-p,p + a + f3 + l;f3+l;x) (4.5) 

for x < 1. 

Only one solution of the equation £)/ = Xf is contained in L 2 at infinity, it 
has the form 



const -F[p + a + l,p + a + (3 + 1; 2p + a + /3 + 2; l/x]x" 



-a — /3—p— 1 



on [1, oo]. Thus, on the both segments the eigenfunction 5 coincides with <J> p up 
to scalar factors. The gluing condition is (|3.9(l . By the last remark of Subsection 

PQJ p - 6 e z. 

if 

Rep = -(a + /3 + l)/2 (4.6) 

then there is no L 2 -eigenfunctions at infinity. □ 

4.3. Self-adjointness. By the previous lemma, the equations D* f = ±if 
have no solution in r K. This implies the essential self-adjointness of D. 

4.4. Specter. The eigenvalues A = — p(p + a + /3 + 1) corresponding to the 
functions $ p form a discrete specter. The remaining specter corresponds to the 
semi-line QUfy . i.e., A > (a + (3 + l) 2 /4. 

Indeed, in all the other cases, we have precisely one L 2 solution Sq(x) of the 
differential equation Df — Xf near 0, and precisely one L 2 -solution Soo(x) near 
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infinity. Hence we can write the Green kernel (i.e., the kernel of resolvent) as it 
is explained in Thus for such A the resolvent exists. 

4.5. Almost L 2 -eigenfunctions. Let 

p = -(a + /3 + l)/2 + is, set 

and A is given by (|4.1|) . 

Lemma. The function ty s given by fl.iy\) is a unique almost L 2 -solution of 
the equation DE — AS. 

Proof. Near x — such solution must have the form (|4.5|l . 

We write the following basis A(s,x), A(—s,x) in the space of solutions of 
the equation Df = Xf, 



A(s,x) = F 



1 + 2is ' x 



x 



-{a+0+l)/2-is 



(4.7) 



The both solutions are almost L 2 . Now we must satisfy the boundary condi- 
tions at x = 1. For this, we expand the 3 solutions (|4.5|) and A(±s; x) near the 
point x = 1 . It remains to write the gluing conditions at x — 1 . The calculation 
is long, its reduced to usage of the complement formula for T and elementary 
trigonometry. We omit this. □ 

The formula for the spectral measure follows from the explicit asymptotics 
of almost L 2 -solutions at oo; this is explained in [3J. 
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